Working minds
Preface
제목에 대해:
There are three senses to the title of this book, Working Minds. One is the notion that Cognitive task analysis is the study of cognition in real-world contexts and professional practice at work. A second is the sense that practitioners of CTA are themselves engaged in the work of studying the mind. The third is the sense of studying minds when they are engaged in successful accomplishment - when things “work.” Taken together, these three senses capture what this book is all about. —loc. 62
20년 역사:
We have been conducting applied studies (which today we call Cognitive task analysis) for about twenty years. … we have done CTA projects in more than a hundred distinct domains and have conducted over a thousand CTA interviews. —loc. 92
책의 목적:
In writing this book, we hope to demystify CTA by showing how it is done. … Also, at an emotional level, we want other researchers and practitioners to be able to enjoy and see value in this type of work. … We agreed from the start that the book would not be a survey of CTA methods, but instead would explain, in as much detail as we could provide, how to conduct a CTA study. … Another reason for writing this book is that we wanted to collaborate with each other. … Finally, another hope is that this is merely the first book of its kind. We do not pretend to have cast any “final words” in stone, or that everything we say here is exactly right.—loc. 109
Acknowledgments
Introduction
Chapter 1. Introduction
Henry Ford와 Nikola Tesla의 일화:
Tesla walked up to a wall of boilerplates, scanned them briefly, and then made an “X” in chalk on one of the plates. Examination of the boilerplate showed that it was indeed faulty. Ford was impressed, and told Tesla to send an invoice. The bill arrived, for $10,000. Ford, never known for his generosity, was astonished at the cost of writing an “X” on the boilerplate, and asked for a breakdown. Tesla sent another invoice, which read:
- Marking wall: $1
- Knowing where to mark: $9,999
This story speaks directly to the purposes and goals of this book in two respects. First, the story illustrates the “why” of Cognitive Task Analysis. What is it that Tesla knows, and how does he know it? What tells him what to do, with Henry Ford looking over his shoulder? Capturing that knowledge and reasoning is one of the things CTA can do. Second, the story illustrates “how” of CTA. CTA can be thought of as a set of tools in a toolkit. Like any tool, CTA can be employed well and wisely, or it can be employed poorly or inappropriately. What tool would you use if you wanted to understand how Tesla was able to grasp the nature of the problem so quickly?
—loc. 164
What CTA offers:
All CTA procedures have the general goal of helping researchers understand how cognition makes it possible for humans to get things done and then turning that understanding into aids—low or high tech—for helping people get things done better. —loc. 176
In all these cases, performance depends on what people know, what they perceive, what they believe, and how they think. —loc. 176
In writing this book, we hope to increase greatly the number of people with the skills and knowledge to conduct high-quality CTA. There are individuals and organizations with problems they cannot solve and opportunities they want to take advantage of. They need CTA tools and methods, and people who know how to apply them skillfully, across a range of problems and issues. —loc. 194
Unpacking CTA:
Cognitive: When the tasks that people are doing are complex, it is not enough to simply observe people’s actions and behaviors. It is also important to find out how they think and what they know, how they organize and structure information, and what they seek to understand better. This is a principal reason why the word “cognitive” begins the phrase CTA.
Task: … in complex cognitive systems, it is not always the literal action sequences that matter as much as the fact that practitioners are trying to get things done…. Therefore, we define task in this broader sense as the outcomes people are trying to achieve.
Analysis: We use the term “analysis” deliberately…. CTA methods provides procedures for systematic, scientific examination to support description and understanding. —loc. 194
Organization of the book:
Part I … provides detailed guidance for planning and carrying out CTA. It includes chapters on capturing knowledge and on capturing the way people reason. We rely on this distinction throughout the book: CTA investigates what people know and how they think.
Part II … provides a perspective on studying cognition in real-world settings and what an expanded view of cognition - a macrocognitive framework - offers. We describe some of the issues that surround CTA and what it means to study cognition in context. We end the section by exploring the challenges of rapidly changing technology.
Part III … describes key issues in applying CTA findings to several applications areas…. We also present a chapter on the role of CTA in the development of measures for evaluating cognitive work.
—loc. 275
Chapter 2. Overview of Cognitive Task Analysis Methods
Primary aspects of CTA:
The three primary aspects of CTA are knowledge elicitation, data analysis, and knowledge representation. Each of these aspects is critical to a successful CTA study. Many people equate CTA with the first aspect, eliciting the knowledge, because traditionally that has received the most attention. But if you don’t do a good job of analyzing your data, why bother collecting them? And if you don’t represent your findings so that others can understand them and why they matter, what have you accomplished? —loc. 288
Knowledge elicitation is the set of methods used to obtain information about what people know and how they know it: the judgments, strategies, knowledge, and skills that underlie performance. —loc. 288
Combination of CTA methods:
… in many CTA projects methods are used in combination. Using various tools and techniques in conjunction provides greater leverage and deeper insight. —loc. 576
Data analysis and knowledge representation:
The analysis phase of CTA is the process of structuring data, identifying findings, and discovering meaning. Knowledge representation includes the critical tasks of displaying data, presenting findings, and communicating meaning…. In much of the CTA literature, analysis and representation are inherently linked to knowledge elicitation and are not treated as separate processes at all…. However, many knowledge elicitation methods produce data that can be analyzed in many different ways and represented using a variety of formats….
- Capsulizing incidents
- Cataloguing cues and patterns
- Identifying themes
- Coding the data
- Describing cognitive sequences
Summary:
… there is not single right way to do CTA. Practitioners of CTA have a wide range of choices in the strategy to use in knowledge elicitation, data analysis, and knowledge representation. —loc. 719
Part I. Tools for Exploring Cognition in Context
Chapter 3. Preparation and Framing
시작하기:
… our advice is to resist the urge to jump into data collection. Experienced CTA researchers take the time to do initial groundwork…. In this chapter we suggest some key issues to think about and activities to carry out during the initial phase of a CTA project. —loc. 673
Framing the CTA project:
Framing the CTA project is the task of sharpening questions, focusing its goals, and identifying any constraints…. Let’s take a look at why framing might matter…. During the discussion, one of the nurses commented, “I’ve heard a good NICU nurse can tell 24 hours in advance of any lab test results that a baby is about to go bad” (Klein 1998). Other nurses in the room nodded…. Good nurses seem to have clear and accurate “intuitions” about infant illnesses. The key question became, “What is ‘nurses’ intuition?’ What does it look like, cognitively and behaviorally?” The study eventually centered on nurses’ perceptual judgments and assessment skills and how their expertise allows them to spot problems early. The frame for the project was about expert clinical assessment skills, but a study of critical care nurses could have been about many other issues. —loc. 693
Here are some suggestions:
- What issue or need do you plan to address?
- What sorts of people can tell you about this issue?
- What aspects of expertise or types of cognition do you need to know about?
- What type(s) of situation(s) will tell you the most about the issue you are exploring?
—loc. 700
Chapter 4. Using Concept Maps for Knowledge Elicitation and Representation
개요
In recent years, a technique called Concept mapping has been adopted by some CTA researchers as a method for both Knowledge elicitation and Knowledge representation. —loc. 925
Background and research foundations
What is concept map?
The concept mapping procedure
Team of group concept mapping
What makes for a good concept map in the CTA context?
Summary
CTA involves capturing what practitioners know about their domain: its concepts, principles, and events. We can think of no CTA process or project in which the CTA researchers did not have to elicit and then represent at least some domain knowledge. This chapter reviews the procedures and applications of concept mapping as a proven knowledge elicitation method for the efficient elicitation of practitioner knowledge. Concept maps involve labeled nodes and links, but concept maps differ in important ways from other types of diagrams that utilize combinations of graphical and textual elements to represent or express meanings. Concept mapping supports the practitioner’s effort to reach for crystal clarity about what he or she wishes to express. In concept-mapping knowledge elicitation, the researchers help the domain practitioner build up a representation of domain knowledge, in effect merging the activity of knowledge elicitation and the activity of knowledge representation. —loc. 1355
Chapter 5. Incident-Based CTA: Helping Practitioners “Tell Stories”
개요
One of the most powerful knowledge elicitation methods available to CTA practitioners s to probe actual incidents. People tell us about all kinds of details, challenges, subtle cues, background influences, and strategies that might never come to light in a general interview or a controlled simulation. Skilled decision makers have had many different experiences; that’s how they formed their knowledge and honed their skills. Their stories can be a doorway into that experience. —loc. 1285
What sorts of things can the CTA researcher find in stories?
- The cues and patterns that experts perceive
- The rules of thumb they have devised
- The kinds of decisions they have to make
- The features that make decisions tough
- The features that make cases typical
- The features of rare cases
The Critical decision method (CDM) Procedure
Boundaries and Limitations of the CDM:
The first condition that limits our ability to do a full CDM is when there simply are no real experts, or even highly skilled practitioners, to be found…. A second condition that can limit the usefulness of CDM is one in which participants are unable to generate useful incidents. Combat-like conditions, where people work under severely stressful conditions and handle very high workloads, can create a blur of events that are difficult to recall as discrete cases. —loc. 1631
Adaptations of CDM
CDM and Here-and-Now Incidents
CDM and Typical Incidents
… not all CTM methods that are incident-based rely on the study of critical incidents…. Sometimes CTA researchers want to understand how things usually or typically work. In other cases, concerns around memory issues may lead the researcher to go after very recent events, to make sure memory for details is fresh. —log. 1673
Variations on Use of a Timeline
Conducting CDM Over Multiple Sessions
The Knowledge Audit as Incident-Based CTA
The most thoroughly tested and validated adaptation of the CDM concept is the Knowledge Audit method (Hutton and Militello 1996; Hutton, Militello, and Miller 1997; Klein and Militello 2004; Militello and Hutton 1998). The Knowledge Audit examines the nature of the expertise needed to perform work skillfully. It structures an interview around a set of probes covering different aspects of expertise. —loc. 1696
Incident-Based CTA with Teams
Summary
In this chapter, we described one method for using incidents to extract cognitive elements - the Critical decision method. We described the procedures for conducting a CDM interview and offered an interviewer’s perspective on each of the CDM components. We examined the boundary conditions under which CDM is less likely to be effective, and we described some of the variations and adaptations that have developed to take advantage of the data collection opportunities that real, lived experience offers. —loc. 1727
Chapter 6. CTA Methods and Experiment-Like Tasks
개요:
Researchers have borrowed or adapted a number of methods from the psychology laboratory to examine cognitive work….
- In constrained processing (CP) methods, familiar tasks or routines are constrained in some way. The participant may be explicitly instructed to adopt a particular strategy, for example. Conversely, the participants may be confronted with a task that challenges their usual strategy.
- In limited information (LI) methods, participants are asked to solve problems given incomplete information.
—loc. 1748
Examples Example 1
In Hoffman’s (1987) study, experts were presented with aerial photos but were allowed only a two-minute inspection period. The experts then had to recall everything they could about the photos and provide their interpretation (e.g., “This region is an arid climate with shallow soils overlying tilted interbedded sandstone and limestone”). At first, the experts balked at the artificiality, but when encouraged to think of the task as a game rather than as a test or a challenge to their expertise, they found the task to be interesting. The task was based on one of the activities involved in the experts’ familiar task, but put a severe restriction on both time (constrained processing) and the amount of information available (limited information).
Results from this task revealed the extent to which the experts seem to achieve immediate perceptual understanding of terrain when viewing aerial photos. For instance, after inspecting the tropical imagery for two minutes one expert commented: “If you were to send troops there they would have to be protected from bacterial infections.” When asked how he knew that, the expert commented that he could tell from the ponds. The expert could see bacteria in a pond from forty thousand feet? No, but what the expert could see was flat interbedded limestone (in a homogeneous forest, the tree canopy informs about the terrain slope) in a tropical climate. Because the bedrock was flat-lying, the streams led into ponds having no major tributary or distributary for an outlet. Stagnant water in a tropical climate means leguminous water plants, implying that the waterways would be laden with bacteria. This all seems like a long inference chain in retrospect, but in the image inspection task, it seemed more a matter of immediate perception built upon a refined base of perceptual learning. —loc. 1791
Example 2
Think aloud problem solving
The TAPS Procedure
Questions about TAPS
Example 1: Coding for task procedures
Example 2: Coding for propositions for a model of knowledge
Example 3: Coding for leverage points
Coding verification
Effort considerations
Variations on the theme of experiment-like tasks
Experiment-like tasks may be useful, for example, in probing the specialized sub-domain knowledge or reasoning of experts (Hoffman 1987) or probing specific hypotheses about reasoning or strategies. They may be less useful if it seems impossible to compose a task that possesses ecological representativeness and validity for the participants who are domain practitioners, but also makes absolutely no sense to novices, apprentices, or journeymen….
We do not see this as an either-or choice. Cognitive Task Analysis conducted in a field setting can involve control and manipulation of variables, just as laboratory research can involve capturing the “real world.” However, the forms of control, selection, and manipulation of variables can be different in a CTA study than in a laboratory setting. Conversely, the ways in which the “real world” is captured in the laboratory result in significant differences from naturalistic studies. Fortunately, we see many ways to use experiment-like manipulations within CTA investigations, as described and illustrated in this chapter….
A combination of strengths would be to use what are perceived to be the more “natural” methods (interviews, observations, etc.) to construct a tentative macrocognitive model (of reasoning, cognition, knowledge, etc.), and then to use methods that are believed to involve more control, selection, and manipulation of variables to study the components of that model. —loc. 2020
Summary:
In this chapter we explored some intersections of laboratory methods and field research methods for conducting CTA. Researchers can tinker with aspects of a familiar task in a variety of ways, thereby eliciting experts’ strategies and reasoning, and do so in structured ways that provide empirical leverage. We also discussed think-aloud problem-solving and protocol methods and the use of analytic and representational formats that can reveal important aspects of cognitive process. —loc. 2020